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ABSTRACT 


Linear  regression  models  are  commonly  used  to  predict  some  criterion  variable  of  interest.  Here,  the  criterion 
variable  of  interest  is  a  grade  assigned  to  Navy  and  Marine  Corps  student  aviators  at  the  end  of  Primary  flight 
training.  It  would  be  of  value  to  be  able  to  predict  such  a  flight  grade  based  on  prior  existing  information.  The 
prior  information  is  inserted  into  a  linear  regression  model  through  some  number  of  predictor  variables.  We  are 
currently  focusing  our  efforts  on  predictor  variables  arising  from  scores  on  the  Aviation  Selection  Test  Battery 
(ASTB),  a  final  grade  from  Aviation  Pre-Flight  Indoctrination  (API),  and  performance  on  a  psychomotor  test 
battery.  We  then  calculated  the  posterior  probability  of  all  possible  linear  regression  models.  Some  models  had  an 
insignificant  posterior  probability  and  we  may  safely  ignore  them  in  any  averaging  over  models.  The  central  result 
of  such  an  exhaustive  analysis  is  the  identification  of  all  those  models,  together  with  their  constituent  predictor 
variables  and  regression  coefficients,  that  do,  in  fact,  possess  a  significant  posterior  probability.  The  predictions  of 
flight  grades  for  any  given  values  of  the  predictor  variables  can  then  be  averaged  over  these  models.  A  major 
advantage  of  this  approach  is  the  assessment  of  the  variability  of  the  predictions.  Of  more  immediate  concern, 
these  results  can  be  incoiporated  into  the  Pilot  Prediction  System  (PPS)  at  the  Naval  Aerospace  Medical  Research 
Laboratory.  The  PPS  is  used  to  make  statistical  inferences  about  the  training  outcome  of  a  student  in  flight  training 
given  information  from  a  data  base.  We  routinely  use  the  PPS  to  make  predictions  about  the  probability  of  failure 
in  some  post- API  phase  of  training  given  scores  from  the  ASTB  and  API.  The  results  from  this  paper  could  be 
inserted  into  the  PPS  to  make  predictions  about  the  flight  grade  in  Primary  training. 
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INTRODUCTION 


Linear  regression  models  are  commonly  used  to  predict  some  criterion  variable  of  interest.  Here,  the  criterion 
variable  of  interest  is  a  grade  assigned  to  Navy  and  Marine  Corps  student  aviators  at  the  end  of  Primary  flight 
training. 

It  would  be  of  value  to  be  able  to  predict  such  a  flight  grade  based  on  prior  existing  information.  The  prior 
information  is  inserted  into  a  linear  regression  model  through  some  number  of  predictor  variables.  We  are  currently 
focusing  our  efforts  on  predictor  variables  arising  from  scores  on  the  Aviation  Selection  Test  Battery  (ASTB),  a 
final  grade  from  Aviation  Pre-Flight  Indoctrination  (API),  and  performance  on  a  psychomotor  test  batteiy. 

In  a  previous  paper  [1],  we  analyzed  the  data  from  210  subjects  who  had  participated  in  the  Naval  Aerospace 
Medical  Research  Laboratory’s  (NAMRL)  version  of  a  psychomotor  test  battery  called  the  CBPT1.  In  that  study, 
we  looked  at  a  total  of  25  predictor  variables.  A  Bayesian  approach  was  employed  to  find  the  posterior 
probabilities  for  a  selected  subset  of  all  possible  linear  regressions  from  the  set  of  25  variables.  Subsequently,  the 
posterior  probability  was  used  as  a  weighting  function  to  average  the  predictions  of  the  flight  grade  for  Primary 
from  each  selected  regression  model.  Rather  than  focusing  on  just  one  model  to  the  exclusion  of  all  others,  the 
Bayesian  model  averaging  approach  averages  the  predictions  of  many  models  to  reduce  the  overall  uncertainty  of 
the  prediction. 

In  this  paper,  we  employ  the  same  technique  of  Bayesian  model  averaging.  However,  we  pare  down  the 
original  list  of  25  predictor  variables  to  a  more  manageable  number  of  eight  predictor  variables.  We  believe  that 
these  eight  new  predictor  variables,  because  they  are  averages  or  composites  of  the  original  25,  are  a  good 
summary  of  the  information  contained  in  the  original  25  predictor  variables. 

With  only  eight  predictor  variables,  it  is  feasible  to  examine  all  possible  linear  regressions.  There  are  256 
possible  models,  ranging  from  the  simplest  model  with  no  predictor  variables  to  the  most  complex  model  with  all 
eight  variables.  Note  that  we  are  not  considering  models  where  predictor  variables  might  enter  as  interactions  with 
other  variables  or  where  the  predictor  variables  are  raised  to  some  power. 

We  then  calculate  the  posterior  probability  of  all  256  linear  regression  models.  Some  models  have  an 
insignificant  posterior  probability,  and  we  may  safely  ignore  them  in  any  averaging  over  models.  The  central  result 
of  such  an  exhaustive  analysis  is  the  identification  of  all  those  models,  together  with  their  constituent  predictor 
variables  and  regression  coefficients,  that  do,  in  fact,  possess  a  significant  posterior  probability.  The  predictions  of 
flight  grades  for  any  given  values  of  the  predictor  variables  can  then  be  averaged  over  these  models.  A  major 
advantage  of  this  approach  is  the  assessment  of  the  variability  of  the  predictions. 

The  net  result  is  that  better  predictions  can  be  made  for  flight  students  entering  Primary  flight  training.  A 
decision  algorithm  could  be  implemented  to  take  advantage  of  these  better  predictions.  Such  a  decision  algorithm 
would  be  an  important  component  in  achieving  significant  cost  savings  for  naval  aviation 

Of  more  immediate  concern,  these  results  can  be  incorporated  into  NAMRL’s  Pilot  Prediction  System  (PPS). 
The  PPS  is  used  to  make  statistical  inferences  about  the  training  outcome  of  a  student  in  flight  training  given 
information  from  a  data  base.  We  routinely  use  the  PPS  to  make  predictions  about  the  probability  of  failure  in 
some  post-API  phase  of  training  given  scores  from  the  ASTB  and  API.  The  results  from  this  paper  could  be 
inserted  into  the  PPS  to  make  predictions  about  the  flight  grade  in  Primary  flight  training. 

ENUMERATION  AND  LISTING  OF  ALL  POSSIBLE  SUBSETS 

Before  we  can  begin  the  analysis,  it  is  necessary  to  systematically  list  all  possible  subsets  from  the  total  of 
eight  predictor  variables.  The  grand  total  of  all  possible  subsets  is  28  =  256  subsets.  We  use  the  combinatorial 
formula  for  choosing  k  objects  from  a  total  of  n  objects  to  find  the  number  of  subsets  with  k  predictor  variables  as 

1CBPT  stands  for  Computer  Based  Psychomotor  Tests. 
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chosen  from  the  universe  of  n  predictor  variables.  The  combinatorial  formula  is  presented  below  as  Equation  (1). 


f  ri\  nl 
\k)  =  (n  —  k)\  k\ 

Table  1  lists  the  results  from  Equation  (1)  as  k  advances  from  no  predictor  variables  (k  ==  0)  in  the  linear 
regression  model  to  all  the  predictor  variables  (k  =  8)  included  in  the  model. 


(i) 


Table  1:  The  enumeration  of  all  possible  subsets  of predictor  variables  from  a  total  of  8  as  the  number  of predictor 
variables  is  increased  from  0  to  8 . 
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Sum  256 


As  an  example  of  the  use  of  Equation  (1),  see  row  5  of  Table  1  where  we  are  interested  in  all  possible 
combinations  of  k  =  4  predictor  variables  from  the  available  total  of  n  =  8. 


8  •  7  •  6  •  5 
4  •  3  •  2  •  1 

=  70 


We  must  therefore  compute  70  linear  regression  models  in  order  to  take  into  account  all  possible  combinations  of 
four  predictor  variables  from  a  total  of  eight  predictor  variables.  Note  that  the  sum  in  the  third  column  of  Table  1 
equals  256. 

After  calculating  the  number  of  subsets  with  k  predictor  variables,  we  actually  had  to  explicitly  generate  these 
possible  combinations.  For  an  example  of  this  task,  refer  to  Table  2,  which  shows  a  listing  of  all  possible 
combinations  for  k  =  2  predictor  variables.  We  constructed  tables  like  this  for  k  =  0  through  k  —  8.  The  total 
number  of  possible  combinations  of  two  predictor  variables  from  a  total  of  eight  predictor  variables  is 

7  +  6  +  5  +  4  +  34-2+1  =:  28 


which  agrees  with  row  3  in  Table  1. 
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Table  2:  A  listing  of  the  28  possible  combinations  of  two  predictor  variables  from  a  total  of  eight  predictor  variables . 
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THE  CRITERION  VARIABLE  AND  THE  EIGHT  PREDICTOR  VARIABLES 

In  this  section,  we  provide  some  more  details  about  the  criterion  variable  and  the  eight  predictor  variables  that 
are  involved  in  the  linear  regression  models.  The  criterion  variable  is  the  standardized  flight  grade  from  Primary 
flight  training.  This  score  was  designed  to  have  a  mean  of  50  and  a  standard  deviation  of  10.  For  our  sample  of 
210  subjects,  the  mean  flight  grade  was  50.29  and  the  standard  deviation  was  10.82. 

In  subsequent  discussions,  the  eight  predictor  variables  are  referred  to  genetically  as  PI,  P2,  •  •  •  P8.  They  are 
either  the  same  variables  used  in  the  previous  referenced  analysis  or  are  composites  of  these  variables.  PI  is  the 
standardized  final  grade  from  API.  P2  through  P6  are  psychomotor  variables  drawn  from  the  CBPT.  P2  is  the 
tracking  error  when  using  the  stick  alone  as  the  controlling  device.  P3  is  the  tracking  error  when  the  stick,  mdder, 
and  throttle  are  all  used  as  controlling  devices.  P4  is  the  average  over  six  sessions  of  horizontal  tracking  error.  P5 
is  the  average  over  three  sessions  of  a  dual  task  consisting  of  horizontal  tracking  with  an  absolute  difference 
computation.  P6  is  the  average  over  four  sessions  of  a  spatial  orientation  skill  called  the  Manikin  task. 

The  correspondence  between  the  notation  in  this  paper  and  the  previous  analysis  (see  Table  1  of  Reference  [1]) 
is  as  follows:  PI  is  API,  P2  is  PMT2,  P3  is  PMT6,  P4  is  the  average  of  PMT8  through  PMT13,  P5  is  the  average 
of  PMT14  through  PMT16,  and  P6  is  the  average  of  PMT17  through  PMT20.  The  last  two  predictor  variables,  P7 
and  P8,  are  composites  of  four  of  the  subtests  that  make  up  the  ASTB.  These  four  subtests  are  the  Math/Verbal 
Test  (MVT),  the  Mechanical  Comprehension  Test  (MCT),  the  Aviation/Nautical  Interest  Test  (AN1),  and  the 
Spatial  Apperception  Test  (SAT).  P7  is  the  same  as  the  AQR  composite,  which  is  the  following  weighted  average 
of  the  subtests 

AQR  =(3x  MVT  )  +  (2  x  MCT  )  +  (2  x  ANI )  +  (1  x  SAT  ). 

P8  is  the  same  as  the  PFAR  composite,  which  is  the  weighted  average  of 

PFAR  =  (1  x  MVT  )  +  (1  x  MCT  )  +  (4  x  ANI )  +  (3  x  SAT  ). 

The  AQR  is  designed  to  correlate  with  academic  performance  during  training,  while  the  PFAR  is  designed  to 
correlate  with  pilot  flight  aptitude. 

THE  POSTERIOR  PROBABILITY  FOR  ALL  256  MODELS 


In  reference  [1],  we  presented  in  detail  the  Bayesian  formalism  for  model  evaluation.  The  outcome  was  a 
formula  for  computing  the  posterior  probability  for  any  given  model.  For  the  kth  model,  the  posterior  probability  is 


P{Mk\D)  = 


exp  (1/2  BICk) 
Ef=i  exp  (1/2  BICi) 


(2) 
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where  BICk  stands  for  the  Bayesian  Information  Criterion  (BIC)  of  the  fcth  model.  It  is  a  function  of  the  squared 
sample  multiple  correlation  coefficient  fR2),  the  sample  size  (n  =  210),  and  the  number  of  predictor  variables 
(p  —  0  to  8)  included  in  the  Unear  regression  model.  Rafteiy  [2]  has  shown  that  the  BIC  is  a  reasonable 
approximation  to  the  posterior  odds  for  linear  regression  models. 

In  Appendix  A,  we  present  the  posterior  probability  for  all  256  regression  models.  This  appendix  is  an  EXCEL 
spreadsheet  and  contains  the  necessary  components  for  computing  the  posterior  probability  according  to  Equation 
(2).  All  models  with  posterior  probability  greater  than  .01  are  highlighted  in  gray. 

Only  a  few  of  the  256  models  possess  any  significant  posterior  probability.  The  model  with  by  far  the  largest 
probability  at  P(Mk\D)  =  .3633  is  the  model  consisting  of  four  predictor  variables,  PI,  P2,  P7,  and  P8.  This 
corresponds  to  the  model  consisting  of  the  API  score,  the  simple  psychomotor  tracking  variable,  AQR  and  PFAR. 
The  model  with  the  next  largest  probability  is  a  model  with  just  two  predictor  variables,  PI  and  P2.  The 
probability  for  this  model  is  P(Mk\D)  =  .1821.  This  corresponds  to  the  model  consisting  of  the  API  score  and 
the  simple  psychomotor  tracking  variable.  Next  in  Une  as  the  third  largest  model  with  P(Mk\D)  =  .0595  is  the 
model  consisting  of  PI,  P2,  and  P7.  This  corresponds  to  the  model  with  the  API  score  and  the  simple  psychomotor 
tracking  variable  as  in  the  previous  two  models,  together  with  a  third  predictor  variable,  AQR. 

BAYESIAN  MODEL  AVERAGING 

There  are  only  15  models  with  a  probabihty  greater  than  .01.  Appendix  B  lists  these  top  15  models  together 
with  their  associated  probabilities.  This  appendix  is  divided  into  two  parts;  the  top  half  consists  of  10  columns  and 
the  bottom  half  consists  of  5  columns. 

In  the  top  half,  the  first  column  contains  the  model  number  of  the  15  top  ranked  models.  The  next  nine 
columns  contain  the  regression  coefficients  for  the  constant  and  the  eight  predictor  variables  for  the  particular 
model  as  given  in  the  first  column. 

The  bottom  half  begins  by  repeating  the  listing  of  the  top  15  linear  regression  models.  In  the  next  column  is 
given  the  prediction  of  the  criterion  variable,  the  standardized  Primaiy  flight  grade.  The  third  column  contains  the 
adjusted  posterior  probability  for  each  model.  This  adjustment  arises  for  the  following  reason.  The  posterior 
probability  for  all  256  models  must  add  up  to  1.  Check  the  final  row  of  Appendix  A  to  verify  this  fact.  However, 
we  are  selecting  only  15  of  these  256  models.  Together,  the  posterior  probability  for  these  15  models  adds  up  to 
.9125,  so  we  must  normalize  the  original  posterior  probability  by  dividing  by  .9125.  This  adjustment  results  in  the 
number  shown  under  P(Model). 

The  next  column  is  the  result  of  multiplying  the  criterion  variable  as  predicted  by  the  regression  model  by  the 
adjusted  posterior  probability.  When  these  values  are  summed,  we  obtain  the  Bayesian  model  average.  The  final 
column  is  the  same  exercise  for  the  standard  deviations  of  the  predictions.  The  mean  value  is  subtracted  from  each 
prediction  and  then  squared.  This  value  is  then  multiplied  by  the  adjusted  probability  and  then  summed.  The 
square  root  of  this  sum  then  yields  the  standard  deviation. 

Data  for  a  numerical  example  of  the  prediction  that  would  be  made  by  this  technique  are  presented  at  the  very 
top  of  Appendix  B.  This  candidate  had  an  API  NSS  of  48,  a  simple  tracking  score  of  25,  a  complex  tracking  score 
of  23,  an  average  horizontal  tracking  score  of  23,  an  average  dual  task  tracking  score  of  8,  an  average  Manikin 
score  of  70,  a  raw  AQR  score  of  158  and  a  raw  PFAR  score  of  165.  The  Bayesian  Model  Averaging  prediction  for 
Primaiy  flight  grade  given  these  scores  is  47.29  with  a  standard  deviation  of  .98.  Employing  the  usual  Normal 
curve  approximation  for  confidence  intervals,  we  would  say  that  we  are  95%  confident  that  this  candidate  would 
receive  a  Primaiy  flight  grade  within  the  interval  of  45  to  49. 
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Appendix  A 


A  listing  of  all  256  linear  regression  models. 


This  appendix  contains  a  listing  of  all  256  linear  regression  models  in  an  EXCEL  spreadsheet  format.  The  first 
column  shows  the  predictor  variables  in  the  model,  the  second  column  shows  the  number  of  predictor  variables, 
and  the  third  column  shows  the  sample  R2.  The  next  five  columns  compute  the  numerator  for  Equation  (2),  and 
the  final  column  presents  the  posterior  probability  for  each  model.  The  final  row  gives  the  denominator  of 
Equation  (2)  and  the  sum  of  the  posterior  probabilities  over  all  256  models.  This  final  sum  must  equal  1.00. 
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0.879 

-27.08 

10.69 

-16.39 

3.62E+03 

0.0000 

P3.P8 

2 

0.143 

0.857 

-32.41 

10.69 

-21.71 

5.19E+04 

0.0000 

P4,PS 

2 

0.054 

0.946 

-11.66 

10.69 

-0.96 

1.62E+00 

0.0000 

P4,p6 

2 

0.063 

0.937 

-13.67 

10.69 

-2.97 

4.42E+00 

0.0000 

P4.P7 

2 

0.053 

0.947 

-If. 44 

10.69 

-0.74 

1.45E+00 

0.0000 

p4.p8 

2 

0.078 

0.922 

-17.05 

10.68 

-6.36 

2.40E-KD1 

0.0000 

P5,p6 

2 

0.036 

0.964 

-7.70 

10.69 

2.99 

2.24E-01 

0.0000 

p6,p7 

2 

0.024 

0.976 

-5.10 

10,69 

5.59 

6.10E-02 

0.0000 

P5,P8 

2 

0.054 

0.946 

-11.66 

10.69 

-0.96 

1.62E+00 

0.0000 

P6,p7 

2 

0.033 

0.967 

-7.05 

10.69 

3.65 

1.61E-01 

0.0000 

P6.P8 

2 

0.062 

0.938 

-13.44 

10,69 

-2.75 

3.95E+00 

0.0000 

2 

0.039 

0.961 

-8,35 

10.69 

2.34 

3.10E-01 

0.0000 

Bill 

.•  :V 

1 

-1 

ill: 

i  ft! 

I 

P1.P3.P4 

3 

0  305 

0.695 

-76.41 

16,04 

-60.37 

1.28E+13 

0.0013 

Pl.p3.p5 

3 

0,305 

0.695 

-76.41 

16.04 

-60.37 

1.28E+13 

0.0013 

Pl.p3,p6 

3 

0.303 

0.697 

-75.80 

16.04 

-59.76 

9.49E+12 

0.0009 

Pl.p3.p7 

3 

0.312 

0.688 

-78.53 

16.04 

.-6£49 

3.71  E+1 3 

0.0037 

Pl.p3.p8 

3 

0.308 

0.692 

-77.32 

16.04 

-61.27 

2.G2E+1 3 

0.0020 

Pl.P<p5 

3 

0.267 

0.733 

-65.23 

16.04 

-49.19 

4.79E+10 

0.0000 

Pl.p4.p6 

3 

0.268 

0.732 

-65.51 

16.04 

-49.47 

5.53E+10 

0.0000 

Pl.p4,p7 

3 

0.280 

0.720 

-68.99 

16.04 

-52.94 

3.14E+11 

0.0000 

Pl,p4.p8 

3 

0.271 

0.729 

-66.38 

16.04 

-50.34 

8.52E+10 

0.0000 

Pl.p5.p6 

3 

0.246 

0.754 

-59.30 

16.04 

-43.25 

2.47E+09 

0.0000 

Pl.p5.p7 

3 

0.256 

0.744 

-62.10 

16.04 

-46.06 

1.00E+10 

0.0000 

Pl.p5.p8 

3 

0.248 

0.752 

-59.85 

16,04 

-43.81 

3.26E+09 

0.0000 

Pl.p6.p7 

3 

0,257 

0.743 

-62.38 

16.04 

-46.34 

1.16E+10 

0.0000 

Pl.p6.p8 

3 

0.251 

0.749 

-60.69 

16.04 

-44.65 

4.97E+09 

0.0000 

Pl.p7.p8 

3 

0.286 

0.714 

-70.74 

16.04 

-54.70 

7.56E+11 

0.0001 

A-3 


MODEL 

P 

RA2 

1  -RA2 

n*ln(1-RA2) 

p  In  n 

BIC 

exp(-.5*BIC) 

P(Model) 

p2,p3,p4 

3 

0.135 

0.865 

-30.46 

16.04 

-14.41 

1.35E+03 

0.0000 

p2,p3,p5 

3 

0.134 

0.866 

-30.21 

16.04 

-14.17 

1.19E+03 

0.0000 

p2,p3,p6 

3 

0.135 

0.865 

-30.46 

16.04 

-14.41 

1.35E+03 

0.0000 

p2,p3,p7 

3 

0.139 

0.861 

-31.43 

16.04 

-15.39 

2.19E+03 

0.0000 

p2,p3,p8 

3 

0.159 

0.841 

-36.36 

16.04 

-20.32 

2.59E+04 

0.0000 

p2,p4,p5 

3 

0.117 

0.883 

-26.13 

16.04 

-10.09 

1.55E+02 

0.0000 

p2,p4,p6 

3 

0.121 

0.879 

-27.08 

16.04 

-11.04 

2.50E+02 

0.0000 

p2,p4,p7 

3 

0.121 

0.879 

-27.08 

16.04 

-11.04 

2.50E+02 

0.0000 

p2,p4,p8 

3 

0.141 

0.859 

-31.92 

16.04 

-15.88 

2.80E+03 

0.0000 

p2,p5,p6 

3 

0.115 

0.885 

-25.66 

16.04 

-9.61 

1.22E+02 

0.0000 

p2,p5,p7 

3 

0.115 

0.885 

-25.66 

16.04 

-9.61 

1.22E+02 

0.0000 

p2,p5,p8 

3 

0.137 

0.863 

-30.94 

16.04 

-14.90 

1.72E+03 

0.0000 

p2,p6,p7 

3 

0.120 

0.880 

-26.85 

16.04 

-10.80 

2.22E+02 

0.0000 

p2,p6,p8 

3 

0.141 

0.859 

-31 .92 

16.04 

-15.88 

2.80E+03 

0.0000 

p2,p7,p8 

3 

0.137 

0.863 

-30.94 

16.04 

-14.90 

1.72E+03 

0.0000 

p3,p4,p5 

3 

0.119 

0.881 

-26.61 

16.04 

-10.57 

1.97E+02 

0.0000 

p3,p4,p6 

3 

0.121 

0.879 

-27.08 

16.04 

-11.04 

2.50E+02 

0.0000 

p3,p4,p7 

3 

0.124 

0.876 

-27.80 

16.04 

-11.76 

3.58E+02 

0.0000 

p3,p4,p8 

3 

0.145 

0.855 

-32,90 

16.04 

-16.86 

4.57E+03 

0.0000 

p3,p5,p6 

3 

0.118 

0.882 

-26.37 

16.04 

-10.33 

1 .75E+02 

0.0000 

p3,p5,p7 

3 

0.121 

0.879 

-27.08 

16.04 

-11.04 

2.50E+02 

0.0000 

p3,p5,p8 

3 

0.143 

0.857 

-32.41 

16.04 

-16.37 

3.58E+03 

0.0000 

p3,p6,p7 

3 

0.123 

0.877 

-27.56 

16.04 

-11.52 

3.17E+02 

0.0000 

p3,p6,p8 

3 

0.145 

0.855 

-32.90 

16.04 

-16.86 

4.57E+03 

0.0000 

p3,p7,p8 

3 

0.146 

0.854 

-33.14 

16.04 

-17.10 

5.17E+03 

0.0000 

p4,p5,p6 

3 

0.064 

0.936 

-13.89 

16.04 

2.15 

3.41  E-01 

0.0000 

p4.p5.p7 

3 

0.057 

0.943 

-12.32 

16.04 

3.72 

1.56E-01 

0.0000 

p4.p5.p8 

3 

0.082 

0.918 

-17.97 

16.04 

-1.93 

2.62E+00 

0.0000 

p4,p6,p7 

3 

0.066 

0.934 

-14.34 

16.04 

1.70 

4.27E-01 

0.0000 

p4.p6.p8 

3 

0.091 

0.909 

-20.04 

16.04 

-3.99 

7. 37  E +00 

0.0000 

p4,p7,p8 

3 

0.083 

0.917 

-18.20 

16.04 

-2.15 

2.94E+00 

0.0000 

p5,p6,p7 

3 

0.038 

0.962 

-8.14 

16.04 

7.91 

1.92E-02 

0.0000 

p5,p6,p8 

3 

0.067 

0.933 

-14.56 

16.04 

1.48 

4.78E-01 

0.0000 

p5,p7,p8 

3 

0.062 

0.938 

-13.44 

16.04 

2.60 

2.73E-01 

0.0000 

p5.p7.p8 

3 

0.067 

0.933 

-14.56 

16.04 

1.48 

4.78E-01 

0.0000 

p1,p2,p3,p4 

4 

0.328 

0.672 

-83.47 

21.39 

-62.09 

3.03E+13 

0.0030 

p1,p2,p3,p5 

4 

0.329 

0.671 

-83.79 

21.39 

-62.40 

3.55E+13 

0.0035 

p1,p2,p3,p6 

4 

0.327 

0.673 

-83.16 

21.39 

-61.77 

2.59E+13 

0.0026 

p1,p2.p3,p7 

4 

0,336 

0,664 

-85,99 

21.39  : 

-64,60 

107E+14 

0,0106 

p1.p2,p3,p8 

. 4 . 

0.331 

0.669 

-84.41 

21.39 

-63.03 

4.85E+13 

0.0048 

p1,p2,p4,p5 

4 

0.325 

0.675 

-82.54 

21 .39 

-61.15 

1.90E+13 

0.0019 

p1,p2,p4,p6 

4 

0.323 

0.677 

-81.92 

21.39 

-60.53 

1.39E+13 

0.0014 

p1,p2,p4,p7 

4 

0.334 

0.666 

-85.36 

21.39 

-63.97 

7.78E+13 

0.0077 

p1,p2,p4,p8 

4 

0.326 

0.674 

-82.85 

21.39 

-61.46 

2.22E+13 

0.0022 

p1,p2,p5,p6 

4 

0.321 

0.679 

-81.30 

21.39 

-59.91 

1.02E+13 

0.0010 

p1,p2,p5,p7 

4 

0.330 

0.670 

-84.10 

21.39 

-62.71 

4.15E+13 

0.0041 

p1,p2,p5,p8 

4 

0.325 

0.675 

-82.54 

21.39 

-61.15 

1.90E+13 

0.0019 

p1,p2,p6,p7 

4 

0.330 

0.670 

-84.10 

21.39 

-62.71 

4.15E+13 

0.0041 

pi  ,p2,p6,p8 

4 

0.324 

0.676 

-82.23 

21.39 

-60.84 

1.63E+13 

0.0016 

p1,p2,p7,p8 

4 

0.358 

a  0.642 

-93  07 

21.39 

-71.68 

3.67E+15 

0.3633 

p1,p3,p4,p5 

. 4 . 

0.308 

0.692 

-77.32 

21.39 

-55.93 

1.39E+12 

0.0001 

p1.p3.p4.p6 

4 

0.305 

0.695 

-76.41 

21.39 

-55.02 

8.85E+1 1 

0.0001 

p1,p3,p4,p7 

4 

0.315 

0.685 

-79.45 

21.39 

-58.06 

4.06E+12 

0.0004 

p1.p3.p4.p8 

4 

0.309 

0.691 

-77.62 

21.39 

-56.23 

1.62E+12 

0.0002 

p1.p3,p5,p6 

4 

0.305 

0.695 

-76.41 

21.39 

-55,02 

8.85E+1 1 

0.0001 

p1,p3.p5,p7 

4 

0.313 

0.687 

-78.84 

21.39 

-57.45 

2.99E+12 

0.0003 

p1,p3,p5,p8 

4 

0.310 

0.690 

-77.92 

21.39 

-56.53 

1.89E+12 

0.0002 

p1.p3.p6.p7 

4 

0.312 

0.688 

-78.53 

21.39 

-57.14 

2.56E+12 

0.0003 

p1,p3,p6,p8 

4 

0.308 

0.692 

-77.32 

21.39 

-55.93 

1.39E+12 

0.0001 

p1,p3,p7,p8  . 

4 

0.342 

0,658 

-S7,9Q  • 

21.39 

-66,51 

2.77E+14 

0,0274 

A-4 


MODEL 

P 

RA2 

1-RA2 

n*ln(1  -RA2) 

p  In  n 

BIC 

exp(-.5*BIC) 

P(Model) 

p1,p4,p5,p6 

4 

0.269 

0.731 

-65.80 

21.39 

-44.41 

4.41  E+09 

0.0000 

p1,p4,p5,p7 

4 

0.281 

0.719 

-69.28 

21.39 

-47.89 

2.51  E+10 

0.0000 

p1,p4,p5,p8 

4 

0.271 

0.729 

-66.38 

21.39 

-44.99 

5.88E+09 

0.0000 

p1.p4.p6.p7 

4 

0.282 

0.718 

-69.57 

21.39 

-48.18 

2.90E+10 

0.0000 

p1,p4,p6,p8 

4 

0.273 

0.727 

-66.95 

21.39 

-45.57 

7.84E+09 

0.0000 

pi  ,p4,p7,p8 

4 

0.314 

0.686 

-79.14 

21.39 

-57.76 

3.48E+12 

0.0003 

p1,p5,p6,p7 

4 

0.259 

0.741 

-62.95 

21.39 

-41.56 

1.06E+09 

0.0000 

p1,p5,p6,p8 

4 

0.252 

0.748 

-60.97 

21.39 

-39.59 

3.94E+08 

0.0000 

p1.p5.p7.p8 

4 

0.295 

0.705 

-73.41 

21.39 

-52.02 

1.98E+11 

0.0000 

p1,p6,p7,p8 

4 

0.294 

0.706 

-73.11 

21.39 

-51.72 

1.70E+11 

0.0000 

p2,p3,p4,p5 

4 

0.135 

0.865 

-30.46 

21.39 

-9.07 

9.31  E+01 

0.0000 

p2,p3,p4,p6 

4 

0.137 

0.863 

-30.94 

21.39 

-9.55 

1.19E+02 

0.0000 

p2,p3,p4,p7 

4 

0.140 

0.860 

-31.67 

21.39 

-10.28 

1.71  E+02 

0.0000 

p2,p3,p4,p8 

4 

0.160 

0.840 

-36.61 

21.39 

-15.23 

2.02E+03 

0.0000 

p2,p3,p5,p6 

4 

0.136 

0.864 

-30.70 

21.39 

-9.31 

1 .05E+02 

0.0000 

p2,p3,p5,p7 

4 

0.139 

0.861 

-31.43 

21.39 

-10.04 

1.51  E+02 

0.0000 

p2,p3,p5,p8 

4 

0.159 

0.841 

-36.36 

21.39 

-14.98 

1.79E+03 

0.0000 

p2,p3,p6,p7 

4 

0.140 

0.860 

-31.67 

21.39 

-10.28 

1.71  E+02 

0.0000 

p2,p3,p6,p8 

4 

0.161 

0.839 

-36.86 

21.39 

-15.48 

2.29E+03 

0.0000 

p2.p3.p7.p8 

4 

0.145 

0.855 

-32.90 

21.39 

-11.51 

3.16E+02 

0.0000 

p2.p4.p5.p6 

4 

0.121 

0.879 

-27.08 

21.39 

-5.70 

1.72E+01 

0.0000 

p2,p4,p5,p7 

4 

0.121 

0.879 

-27.08 

21.39 

-5.70 

1.72E+01 

0.0000 

p2,p4,p5,p8 

4 

0.142 

0.858 

-32.16 

21.39 

-10.77 

2.18E+02 

0.0000 

p2,p4,p6,p7 

4 

0.125 

0.875 

-28.04 

21.39 

-6.65 

2.78E+01 

0.0000 

p2,p4,p6,p8 

4 

0.146 

0.854 

-33.14 

21.39 

-11.75 

3.57E+02 

0.0000 

p2,p4,p7,p8 

4 

0.144 

0.856 

-32.65 

21.39 

-11.26 

2.79E+02 

0.0000 

p2,p5,p6,p7 

4 

0.120 

0.880 

-26.85 

21.39 

-5.46 

1.53E+01 

0.0000 

p2,p5,p6,p8 

4 

0.142 

0.858 

-32.16 

21.39 

-10.77 

2.18E+02 

0.0000 

p2.p5.p7.p8 

4 

0.140 

0.860 

-31 .67 

21.39 

-10.28 

1.71  E+02 

0.0000 

p2,p6,p7,p8 

4 

0.144 

0.856 

-32.65 

21.39 

-11.26 

2.79E+02 

0.0000 

p3.p4.p5.p6 

4 

0.121 

0.879 

-27.08 

21.39 

-5.70 

1.72E+01 

0.0000 

p3,p4,p5,p6 

4 

0.124 

0.876 

-27.80 

21.39 

-6.41 

2.47E+01 

0.0000 

p3,p4,p5,p7 

4 

0.145 

0.855 

-32.90 

21.39 

-11.51 

3.16E+02 

0.0000 

p3,p4,p6,p7 

4 

0.125 

0.875 

-28.04 

21.39 

-6.65 

2.78E+01 

0.0000 

p3,p4,p6,p8 

4 

0.147 

0.853 

-33.39 

21.39 

-12.00 

4.04E+02 

0.0000 

p3,p4,p7,p8 

4 

0.148 

0.852 

-33.64 

21.39 

-12.25 

4.56E+02 

0.0000 

p3,p5,p6,p7 

4 

0.123 

0.877 

-27.56 

21.39 

-6.17 

2.19E+01 

0.0000 

p3,p5,p6,p8 

4 

0.145 

0.855 

-32.90 

21.39 

-11.51 

3.16E+02 

0.0000 

p3.p5.p7.p8 

4 

0.146 

0.854 

-33.14 

21.39 

-11.75 

3.57E+02 

0.0000 

p3,p6,p7,p8 

4 

0.148 

0.852 

-33.64 

21.39 

-12.25 

4.56E+02 

0.0000 

p4,p5,p6,p7 

4 

0.066 

0.934 

-14.34 

21.39 

7.05 

2.95E-02 

0.0000 

p4,p5,p6,p8 

4 

0.091 

0.909 

-20.04 

21.39 

1.35 

5.09E-01 

0.0000 

p4,p5,p7,p8 

4 

0.089 

0.911 

-19.57 

21.39 

1.81 

4.04E-01 

0.0000 

p4.p6.p7.p8 

4 

0.097 

0.903 

-21.43 

.  21.39 

-0.04 

1.02E+00 

0.0000 

p5,p6,p7,p8 

4 

0.076 

0.924 

-16.60 

21.39 

4.79 

9.12E-02 

0.0000 

p1,p2,p3,p4,p5 

5 

0.331 

0.669 

-84.41 

26.74 

-57.68 

3.35E+12 

0.0003 

p1.p2.p3.p4.p6 

5 

0.328 

0.672 

-83.47 

26.74 

-56.74 

2.09E+12 

0.0002 

p1,p2,p3,p4,p7 

5 

0.338 

0.662 

-86.62 

26.74 

-59.89 

1.01E+13 

0.0010 

p1,p2,p3,p4,p8 

5 

0.331 

0.669 

-84.41 

26.74 

-57.68 

3.35E+12 

0.0003 

p1,p2,p3,p5,p6 

5 

6.329 

0.671 

-83.79 

26.74 

-57.05 

2.45E+12 

0.0002 

p1,p2,p3,p5,p7 

5 

0.337 

0.663 

-86.31 

26.74 

-59.57 

8.62E+12 

0.0009 

p1,p2,p3,p5,p8 

5 

0.333 

0.667 

-85.04 

26.74 

-58.31 

4.58E+12 

0.0005 

p1,p2,p3,p6,p7 

5 

0.337 

0.663 

-86.31 

26.74 

-59.57 

8.62E+12 

0.0009 

p1,p2,p3,p6,p8 

5 

0.331 

0.669 

-84.41 

26.74 

-57.68 

3.35E+12 

0.0003 

p1,p2,p3,p7,p8 

5.;-‘:i 

0.363 

0.637 

-94.71 

26.74 

-67.97 

.  5.75E+14 

0.0570 
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Appendix  B 


The  regression  coefficients  and  constant  for  the  best  15  linear  regression  models. 


This  appendix  contains  the  eight  regression  coefficients  and  the  constant  for  the  best  15  models  listed  in  an 
EXCEL  spreadsheet  format.  If  certain  predictor  variables  do  not  appear  in  a  given  model,  then  the  corresponding 
regression  coefficients  are  listed  as  0.000.  The  top  line  contains  fictitious  data  on  each  of  the  eight  tests.  This  is 
used  in  the  numerical  example  of  the  prediction  of  the  flight  grade  by  each  of  the  top  15  models.  This  is  shown  in 
the  bottom  half  of  the  Appendix  along  with  the  adjusted  posterior  probability  of  each  model  and  the  model’s 
contribution  to  the  overall  average  and  standard  deviation. 


Intentionally  blank. 


Appendix  B 


DATA 

;  48 

25  ; 

|  23  I 

8 

!!1!1! 

llllllll 

Bill!! 

Model 

Constant 

PI 

P2 

P3 

P4 

P5 

P6 

P7 

P8 

1 

P1,P2 

-41.027 

0.781 

2.065 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

2 

P1,P3 

-55.695 

0.742 

0.000 

2.929 

0.000 

0.000 

0.000 

0.000 

0.000 

3 

P1,P2,P3 

-55.949 

0.756 

1.524 

1.282 

0.000 

0.000 

0.000 

0.000 

0.000 

4 

P1.P2.P4 

-46.708 

0.773 

1.892 

0.000 

0.435 

0.000 

0.000 

0.000 

0.000 

5 

P1,P2,P5 

-41.741 

0.789 

2.119 

0.000 

0.000 

-0.085 

0.000 

0.000 

0.000 

6 

P1,P2,P6 

-40.941 

0.780 

2.055 

0.000 

0.000 

0.000 

0.003 

0.000 

0.000 

7 

P1,P2,P7 

-33.026 

0.849 

2.051 

0.000 

0.000 

0.000 

0.000 

“0.058 

0.000 

8 

P1,P2,P8 

“46.482 

0.757 

2.041 

0.000 

0.000 

0.000 

0.000 

0.000 

0.034 

9 

P1,P2,P3,P7 

-47.509 

0.824 

1.535 

1.224 

0.000 

0.000 

0.000 

“0.056 

0.000 

10 

P1,P2,P7,P8 

-41.105 

0.858 

1.946 

0.000 

0.000 

0.000 

0.000 

“0.140 

0.122 

11 

PI  ,P3,P7,P8 

-55.120 

0.820 

0.000 

2.736 

0.000 

0.000 

0.000 

“0.140 

0.126 

12 

P1,P2,P3,P7,P8 

-54.884 

0.834 

1.454 

1.171 

0.000 

0.000 

0.000 

“0.138 

0.121 

13 

P1,P2,P4,P7,P8 

-47.215 

0.853 

1.748 

0.000 

0.497 

0.000 

0.000 

-0.143 

0.121 

14 

P1,P2,P5,P7,P8 

-40.472 

0.855 

1.908 

0.000 

0.000 

0.058 

0.000 

“0.143 

0.124 

15 

P1,P2,P6,P7,P8 

-40.817 

0.854 

1.913 

0.000 

0.000 

0.000 

0.010 

-0.141 

0.122 

Model 

Prediction 

P(Model) 

Average 

SD 

1 

PI  ,P2 

48.09 

0.1996 

9.60 

0.13 

2 

P1,P3 

47.29 

0.0149 

0.70 

0.00 

3 

P1,P2,P3 

47.93 

0.0408 

1.96 

0.02 

4 

P1,P2,P4 

47.70 

0.0219 

1.04 

0.00 

5 

P1.P2.P5 

48.42 

0.0161 

0.78 

0.02 

6 

P1.P2.P6 

48.07 

0.0138 

0.66 

0.01 

7 

P1.P2.P7 

49.81 

0.0652 

3.25 

0.42 

8 

P1.P2.P8 

46.54 

0.0255 

1.19 

0.01 

9 

P1,P2,P3,P7 

49.65 

0.0116 

0.58 

0.06 

10 

P1.P2.P7.P8 

46.74 

0.3981 

18.61 

0.12 

11 

P1.P3.P7.P8 

45.84 

0.0300 

1.38 

0.06 

12 

P1,P2,P3,P7,P8 

46.59 

0.0625 

2.91 

0.03 

13 

P1.P2,P4,P7,P8 

46.23 

0.0449 

2.08 

0.05 

14 

P1,P2,P5,P7,P8 

46.60 

0.0275 

1.28 

0.01 

15 

P1,P2,P6,P7,P8 

46.52 

0.0275 

1.28 

0.02 

0.9999  j 

47.29 

0.98 

B-3 


Reviewed  and  approved. 


JZ^4T>C3-2j 


R.  R.  STANNY,  Ph.D. 
Technical  Director 


This  research  was  sponsored  by  the  Office  of  Naval  Research  under  work  unit  62233N.0330.126-7801. 

The  views  expressed  in  this  article  are  those  of  the  authors  and  do  not  necessarily  reflect  the  official  policy  or 
position  of  the  Department  of  the  Navy,  Department  of  Defense,  nor  the  U.S.  Government. 

Volunteer  subjects  were  recruited,  evaluated,  and  employed  in  accordance  with  the  procedures  specified  in  the 
Department  of  Defense  Directive  3216.2  and  Secretary  of  the  Navy  Instruction  3900.39  series.  These  instructions  are 
based  upon  voluntary  informed  consent  and  meet  or  exceed  the  provisions  of  prevailing  national  and  international 
guidelines 

Trade  names  of  materials  and/or  products  of  commercial  or  nongovernment  organizations  are  cited  as  needed  for 
precision.  These  citations  do  not  constitute  official  endorsement  or  approval  of  the  use  of  such  commercial 
materials  and/or  products. 

Reproduction  in  whole  or  in  part  is  permitted  for  any  purpose  of  the  United  States  Government. 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources, 
gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this 
collection  of  information,  including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson 
Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0 188),  Washington,  DC  20503. 


1.  AGENCY  USE  ONLY  (Leave  blank) 


2.  REPORT  DATE 
1  August  2000 


4.  TITLE  AND  SUBTITLE 

Predicting  Primary  Flight  Grades  by  Averaging 
Over  Linear  Regression  Models:  Part  2. 


6.  AUTHOR(S) 

D.  J.  Blower,  A.O.  Albert,  and  H.P.  Williams 


3.  REPORT  TYPE  AND  DATES  COVERED 


5.  FUNDING  NUMBERS 


B994  03330/0126 
Work  Unit  7903 


7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Naval  Aerospace  Medical  Research  Laboratory 
51  Hovey  Road 
Pensacola  FI  32508-1046 


8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 


NAMRL-1411 


9.  SPONSORING  /MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Office  of  Naval  Research 
800  N.  Quincy  Street 
Arlington,  VA  22217-5660 


11.  SUPPLEMENTARY  NOTES 

Approved  for  public  release;  distribution  unlimited. 


10.  SPONSORING  /MONITORING 
AGENCY  REPORT  NUMBER 


12a.  DISTRIBUTION /AVAILABILITY  STATEMENT 


12b.  DISTRIBUTION  CODE 


13.  ABSTRACT  (Maximum  200  words)  . 

Linear  regression  models  are  commonly  used  to  predict  some  criterion  variable  of  interest.  Here,  the  criterion  variable  of 

interest  is  a  grade  assigned  to  Navy  and  Marine  Corps  student  aviators  at  the  end  of  Primary  flight  training.  It  would  be  of 
value  to  be  able  to  predict  such  a  flight  grade  based  on  prior  existing  information.  The  prior  information  is  inserted  into  a 
linear  regression  model  through  some  number  of  predictor  variables.  We  are  currently  focusing  our  efforts  on  predictor 
variables  arising  from  scores  on  the  Aviation  Selection  Test  Battery  (ASTB),  a  final  grade  from  Aviation  Pre-Flight 
Indoctrination  (API),  and  performance  on  a  psychomotor  test  battery.  We  then  calculated  the  posterior  probability  of 
all  possible  linear  regression  models.  Some  models  had  an  insignificant  posterior  probability  and  we  may  safely  ignore 
them  in  any  averaging  over  models.  The  central  result  of  such  an  exhaustive  analysis  is  the  identification  of  all  those 
models,  together  with  their  constituent  predictor  variables  and  regression  coefficients,  that  do,  in  fact,  possess 
a  significant  posterior  probability.  The  predictions  of  flight  grades  for  any  given  values  of  the  predictor  variables  can  then 
be  averaged  over  these  models.  A  major  advantage  of  this  approach  is  the  assessment  of  the  variability  of  the  predictions. 
Of  more  immediate  concern,  these  results  can  be  incorporated  into  the  Pilot  Prediction  System  (PPS)  at  the  Naval 
Aerospace  Medical  Research  Laboratory.  The  PPS  is  used  to  make  statistical  inferences  about  the  training  outcome  of  a 
student  in  flight  training  given  information  from  a  data  base.  We  routinely  use  the  PPS  to  make  predictions 
about  the  probability  of  failure  in  some  post-API  phase  of  training  given  scores  from  the  ASTB  and  API.  The  results  from 
this  paper  could  be  inserted  into  the  PPS  to  make  predictions  about  the  flight  grade  in  Primary  training. 


14.  SUBJECT  TERMS 

Keywords:  aviation  selection,  linear  regression,  model  averagmg,  Bayesian 
statistics,  primary  flight  grades 


15.  NUMBER  OF  PAGES 

24 


16.  PRICE  CODE 


17.  SECURITY  CLASSIFICATION 
OF  REPORT 

UNCLASSIFIED 


I.  SECURITY  CLASSIFICATION 
OF  THIS  PAGE 

UNCLASSIFIED 


.  SECURITY  CLASSIFICATION 
OF  ABSTRACT 

UNCLASSIFIED 


20.  LIMITATION  OF  ABSTRACT 


NSN  7540-01-280-5500 


Standard  Form  298  (Rev.  2-89) 

Prescribed  by  ANSI  Std.  Z39-18 
298-102 


